Bulletin of Electrical Engineering and Informatics 
Vol. 9, No. 5, October 2020, pp. 1835~1843 
ISSN: 2302-9285, DOI: 10.1159 1/ee1.v915.2264 O 1835 


Improved support vector machine using optimization 
techniques for an aerobic granular sludge 


Nur Sakinah Ahmad Yasmin’, Norhaliza Abdul Wahab’, Aznah Nor Anuar” 
'*School of Electrical Engineering, Faculty of Engineering, Universiti Teknologi Malaysia, Malaysia 
*Department of Environment and Green Technology, Malaysia-Japan International Institute of Technology, Malaysia 


Article Info ABSTRACT 

Article history: Aerobic granular sludge (AGS) is one of the treatment methods often used in 
wastewater systems. The dynamic behavior of AGS is complex and hard to 

Received Dec 18, 2019 predict especially when it comes to a limited data set. Theoretically, support 

Revised Mar 7, 2020 vector machine (SVM) is a good prediction tool in handling limited data set. 

Accepted Apr 13, 2020 In this paper, an improved SVM using optimization approaches for better 


predictions is proposed. Two different types of optimization are built which 
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Aerobic granular sludge verification purposes. The experimental data under hot temperature of 50°C 


Modelling obtained from sequencing batch reactor is used. From simulation results, 

Support vector machine the proposed SVM with optimizations improve the prediction of chemical oxygen 

SVM-GA demand compared to the conventional grid search method and hence provide 

SVM-PSO better prediction of effluent quality using AGS wastewater treatment systems. 
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1. INTRODUCTION 

Wastewater treatment is a process of converting wastewater into an effluent before it can be safely 
returned to the water cycle. It is full of contaminants including toxins, bacteria and harmful chemicals that 
are not safe to be consume. The main goals of wastewater treatment is to eliminate all the contaminants 
and ensure the water is safe to discharge back to the water cycle. As the demands of urban area grows 
rapidly, the wastewater treatment nowadays has various kind of water treatment ranging from low price to 
highly cost operations. Compared with drawbacks of the system such as required large space area for 
separation aeration and settling tanks, produced low biomass concentration in aeration tank but high energy 
consumption and operates in space rather than time of conventional activated sludge system, a new 
technology using sequencing batch reactor (SBR) is developed. 

New technology which relies on the growth of aerobic granular sludge (AGS) can helps to 
overcome the limitation of conventional activated sludge system. It has been proved that the formation of 
AGS has a short settling time when inoculated with activated sludge and operate using SBR [1]. Since then, 
AGS technology has been attracting attention among the researcher for its great potential in reducing 
footprints while increasing the efficiency of wastewater treatment plants. AGS technology has been one of 
the promising technologies in biological treatment. Some of the advantages of AGS technology are has 
a compact and regular structure of granules make AGS able to withstand the high load of toxic compound, 
high biomass retention and have a good settle ability [2]. This process of aerobic granulation can be seen 
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happened in SBR. Many studies have extensively investigated the fundamental of AGS in SBR [3-5]. 
A study to identify the most vital feature that influencing the development of AGS process for instance 
settling time, loading rate, hydrodynamic shear force and substrate composition [6] 1s demonstrated in 
a lab scale. 

The formation of AGS is a complex biological process and it influenced by other parameters that 
need to be further examined. Generally the study of AGS using SBR system was performed at 20°C until 
25°C or lower [6, 7]. Hence, it may not suitable for the country with high climate weather temperature such 
as Saudi Arabia. Further investigation is needed to study the effect on AGS process at three different high 
temperature which are 30°C, 40°C and 50°C by using SBR system. However, AGS is a very complex 
occurrence of microbial immobilization [8]. There are numerous internal interaction among the process 
between the variables such as growth and sludge characteristic including oxygen transfer, biomass 
detachment and diffusion. Thus, a comprehensive mathematical model that can takes all these parameters 
into consideration is needed in order to establish a good model. As a highly approach for the limited data set, 
Support Vector Machine has been proposed to develop the model [9]. Many field of studies has been applied 
SVM such as in prediction [10], pattern recognition problem [11], classification [12], regression [13, 14] 
and time series [12, 15]. In addition, SVM has been reported outperformed traditional statistical learning method 
such as artificial neural network. Thus, more studies are focusing on this SVM technique as SVM has been 
arousing more. 

In wastewater treatment, SVM was utilized widely in modelling process. The example of 
implementation of SVM technique on limited dataset can be found in [16], which SVM model is developed 
in order to study the relationship between membrane bioreactor and the affecting factors of fouling. 
This study reported that SVM technique can assure a good prediction results compared to neural network 
model since the depending of the training dataset for SVM technique is lesser than neural network. 
In wastewater treatment, the same method was observed, [17] implement SVM technique to model effluent 
quality with cyclic activated sludge process. This study reported to gives high prediction accuracy with small 
learning ability and good generalization. Other than that, [18] presented a successful SVM technique that was 
applied to predict the permeate flux and rejection of bovine serum albumin. 

The results shows SVM outperformed artificial neural network (ANN) due to ANN technique needs 
to rely on the one experience and expertise while SVM technique is rely on statistical theory. 
The implementation by [19] in predicting the different sample size of membrane permeate flux dead end 
microfiltration of activated sludge reactor also shows SVM model outperformed as compared to back 
propagation neural network. This study shows that small sample size of SVM yields higher accuracy. 
This is because of the rigorous foundation of mathematical and theoretical of SVM despite needs to rely on 
the designer knowledge itself. 

In SVM algorithm, two parameters are involved which are cost, C and gamma, 7. It important to 


select best pair of these two parameters as it will affect the performance of the results. Common method used 
usually are trial and error method or by using grid search. However, these types of method is a conventional 
way and its time consuming [20]. Hence, this paper implement an optimization of a SVM model by using 
genetic algorithm (GA) and particle swarm optimization (PSO). The main goal of this paper is to model 
and optimize the SVM model by using limited dataset. The fitness function used in this work is to minimize 
the mean square error. In this study, the 60 days data which contains of 21 samples under the temperature of 
50°C is utilized to develop the model of AGS by using SBR systems. As for this paper, it is consist of five 
main sections and are organized as follows. The first section started with the background of this study 
and several literature on the proposed previous method. Then, second section will be presented modelling of 
SVM with the proposed optimization method which are GA and PSO. Third section is the detailed of 
the experimental setup of this study while in the fourth section is the results and discussion. Finally, the fifth 
section gives conclusion and future works based on this study. 


2. RESEARCH METHOD 

SVM was first introduced by Vapnik (1995) [13]. The idea of SVM is based on structural risk 
management. Basically, SVM is a supervised learning method that is capable in generating predictive outputs 
of the model and was used for classifying unseen patterns into their categories. Recently, SVM technique has 
been extended its function in solving function estimation and regression problems. The SVM technique 
is based on the concept of optimal hyper lane that separates sample of two different classes by considering 
the widest gap between them. SVM technique solves a quadratic problem in which the objective function in 
obtained by combination of regularization term and loss function. Support vector regression basically 
is an extension of SVM regression. The main purpose of SVM regression is to find a function that has a most 
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y deviation from the actually obtained objectives for all training dataset and at the same time it would be as 
flat as possible. 


2.1. Support vector machine 
The general theory of SVM regression function is expressed as follows in (1). 


y=f(x) =ø@øx+b (1) 


where scalar output is denoted as y, weight vector is denoted as w, while vector and bias is denoted as x 
and b, respectively. These coefficients are estimated by introducing slack variables, €; and €;, to minimizing 
the regularized risk function. These variables represent the distance from the actual values to 
the corresponding boundary value of €-tube. The SVM regression function can be expressed by (2) and (3). 


sb eo 1 f 
Minimize (w) = = loll? =) (<6 20 (2) 
where 


y-wx-b <é+&, 
Subject to; wx —b-—y set ¢,i=1,2,.., (3) 
iS; = 9, 


The constant C is the cost factor and determines the trade of between approximation error 
and weight factor. yis the radius of the tube within where the regression function must lie. Different kernel 


function have been used in SVM regression problems such as linear kernel, polynomial kernel and Gaussian 
kernel. Each type of kernel function has its own parameters which will affect the flexibility of regression 
function. Radial basis function (RBF) is one of those popular kernel function in SVM that has been widely 
used in different types of studies. When there is no prior knowledge about data characteristics, RBF kernel 
is recommended as RBF kernel is intent to give good performance under smoothness assumption. 
After calculating g and g7, thus, the new generated predicted output of SVM regression function for each 


x'is can be given as (4) [21]. 


L 
y'= X (af - a7) x’ +b (4) 
i=1 


2.2. Genetic algorithm 

GA is based on evolutionary algorithms. The basic concept of GAs is designed to simulate processes 
in natural systems especially for evolution. These algorithms are based on survival principle of the fittest 
member in population which tries to retain genetic information from generation to generation [22]. One of 
the GAs advantage characteristics is their inherent parallelism regarding others algorithm that are serial. 
It would eliminated one path turns out to be dead end and other promising paths will be continued. 
Other significant capability of GAs is in finding global optimum among many local optima even on 
the condition of complex problems [23]. 


2.3. Particle swarm optimization 

Basically, PSO is similar to GA since PSO itself is an evolutionary computation technique. PSO 
is based on population searching technique that utilize the idea of social sharing informations. In PSO, 
particles are flown through hyper dimensional search space. The changes of the particles position are based 
on social psychological tendency which are the particles are following the success of other particles. 
Thus, the changes of a particles within the swarm are influence by experience and knowledge of its 
neighbour. Consequently, the behaviour of particles is also affected by that of other particles within 
the swarm. This results in the prediction of a model since that the social behaviour of the particles 
stochastically return toward previously successful areas in the search space. The process of PSO and GA 
optimization are describe in the flow chart in Figure 1. 


Improved support vector machine using optimization techniques... (Nur Sakinah Ahmad Yasmin) 


1838 O 







Define: 

® Initial range of C, £0 
è Size of population 

© Raa, Fam, Pan 

æ Maximum generations 


Create initial population 


Perform SVM on each 
chromosome in population 
and calculate the A/S Fino 






<—_ Max. Generation? “> 


Create next population: 

è Rank fitness scaling 

# Roulette wheel selection 
e Heuristic crossover 


è Uniform mutation 


w 


( Optimal C, £, a for SVM ) 


« Optimization of C. ¢, a 





en i En 
(Start PSO-SVM ) 


— 


Define: 

è Initial range of C, £, o 
è Quantity of particles 
+ Cis Cr W, Vans 

è Maximum interations 


Initialize: 
+ Position of each particle 
® Velocity of each particle 


Perform SVM on each 
particle in population and 
compute the ASS Fico 


Renew p and g 





Yes e O 
<__Marx. Interation? — 


Noy 
Update: 
® Velocityof particles Eq.(14) 
® Position of particles Eq.(15) 





Figure 1. SVM-PSO/GA procedure 


3. EXPERIMENT SETUP 
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In this study, the experimental setup was carried out in the double-walled cylindrical column of 
SBR. The height and internal diameter of SBR are 100 cm and 6.5 cm, respectively. All the operation such as 
influent reading, effluent discharge and aerator pump was controlled by programmable logic controller 
system with the setting time for each phase. The working temperature of SBR were controlled using water 
bath sleeves and thermostat at three different temperatures which are 30°C, 40°C and 50°C + 1°C. In this 
study, the AGS bioreactor operated at 50°C + 1°C is considered. The influent parameters of AGS bioreactor 
that used as input of the model are chemical oxygen demand (COD), total phosphorus (TP), total nitrogen 
(TN), ammonia nitrogen (AN) and mixed liquor suspended solid (MLSS). The output parameter of the model 


is the effluent COD. Figure 2 shows the schematic diagram of the AGS of SBR system. 
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Figure 2. Schematic diagram of laboratory scale AGS bioreactor 
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4. RESULT AND DISCUSSION 

In this section, the modelling and optimization results of AGS are presented. As mention in Section 
3, only bioreactor of 50°C + 1°C is considered. The collection of the real experimental data is limited to only 
21 dataset. Hence, the data was split using 10-fold cross validation (CV) as it was suggested for limited 
dataset [24]. The first 53% from the experimental data was used as the training dataset while the remaining 
47% is used as testing dataset. The 10-CV is then combined with grid search method in order to choose 
the pair of parameters of Cand y. The CV operated by using 9-fold from 53% of training dataset to train 


and leave one last fold as a test dataset. The training model was repeated for ten times and the average 
performance are calculated. Then it will tested again on the 47% to verify the perfomance. The results 
are measured using root mean square error (RMSE) performance criteria. 


4.1. Validation and evaluation of the SVM-GA model 

In statistic, it is important to check the validity of the regression model. The validation of the SVM 
regression model involve by analysing the goodness fit of the model, checking the residual plots plot 
are random and by checking the performance of prediction model towards unknown data (testing data) if 
the performance deteriorates substantially when applied. In order to accurately and objectively map 
the precision of the model, the prediction result of the effluent COD concentration are validated based on 
statistical parameter such as highest correlation coefficient (R^, and lower mean square error (MSE) values. 
The mathematical formulas are given as (5) and (6) [25]. 


wi; -$ 
R? =|-2 (5) 


> w,(y, -5 
iA 


Where $, denoted as prediction data, y, as the mean of actual data 


wy; — 5) 
MSE = =! 
n— m 


(6) 


Where y;, as actual value from the measurement data, n is the number of responses minus the number of 
fitted coefficients m estimated from the response value. Figure 3(a), Figure 4(a) and Figure 5(a) present 
the cross plot between the measured and predicted results for SVM PSO, SVM-GA and SVM Grid-search 
during testing output, respectively. The plots have obtained coefficients values of 94.59%, 94.11% 
and 94.65% during testing phase. These values demonstrate high accuracy of each models as it is closed to 
unity. Although the high coefficient indicate a good model, but there is a need to confirm stochastic nature of 
any intrinsic errors. Hence, residuals is plotted for this purpose. As shown in Figure 3(b), Figure 4(b) 
and Figure 5(b) the residuals are evenly dispersed around zero line (x-axis). 


Cross Plot — Training SBR50°C Residual Plot — Testing SBR50°C 
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Figure 3.(a) Cross plot of SVM-PSO during testing phase for SBR500C (b) Residuals plot of the SVM-PSO 
model’s relative error deviations during testing phase for SBR50°C 
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Figure 4.(a) Cross plot of SVM-GA during testing phase for SBR50°C (b) Residuals plot of the SVM-GA 
model’s relative error deviations during testing phase for SBR50°C 
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Figure 5.(a) Cross plot of SVM-Grid Search during testing phase for SBR50°C (b) Residuals plot of the 
SVM-Grid search model’s relative error deviations during testing phase for SBR50°C 


The RMSE values obtained are 0.07123, 0.07569 and 0.0704, respectively which closed to zero. 
Hence, these results are the indicative of high performing model that validate the generalizability 
and accuracy of the models achieved. The overall summarized results of the cross and residuals plot of 
SVM-PSO, SVM-GA and SVM-Grid Search are shown in Table 1. 


Table 1. Statistical performance for SVM-PSO, SVM-GA and SVM-Grid search model 


Statistical Parameters SVM-PSO SVM-GA SVM-Grid Search 

SBRS50°C Training Testing Training Testing Training Testing 
SSE 0.01781 0.04567 0.01538 0.05156 0.02417 0.04463 

R’ 0.9783 0.9459 0.9819 0.9411 0.9715 0.9465 
Adjusted R? 0.9755 0.9399 0.9796 0.9345 0.9680 0.9405 
RMSE 0.04719 0.07123 0.04384 0.07569 0.05497 0.0704 

P1 0.8593 0.8529 0.8764 0.8665 0.8722 0.8481 
P2 0.04454 0.06321 0.04409 0.06449 0.0416 0.06843 


4.2. Development of SVM grid search model 

Generally, common method usually used in determine the parameters used in SVM are using trial 
and error or grid search method. To make a clear comparison with optimization method which are PSO 
and GA, grid search algorithm is developed. From the tuning parameters grid search for cost and gamma 
values, the algorithm found the local minima with minimum error RMSE of 0.0128. The pairs values of cost 
and gamma obtained are 4 and 0.125. By using the values of parameters obtained, a graph for training 
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and testing data are plotted as shown in Figure 6. From the plotted graph, it shows that the prediction of 
SVM-Grid search during training model gives good correlation with low MSE value which are 96.17% 
and 0.0041, respectively. To validate the results, the algorithm is then used on testing data (unknown data) 
and the graph obtained the values R’ of 93.43% and MSE of 0.0065. 


Training Data- SVM-Grid Search Model a Testing Data- SVM-Grid Search Model 
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Figure 6. Comparison between measured and predicted of SVM Grid-Search for training and testing data 


4.3. Development of hybrid SVM-PSO and SVM-GA model 

The development of optimization method using PSO and GA are developed using Matlab algorithm. 
Both technique shows good agreement accuracy in predicting the model. The simulation results of fitness 
curve for SVM-PSO and SVM-GA are measured using MSE performance criteria. The closer the value to 
the zero is better for the prediction model. From the iteration, the system found the best local minima for 
SVM-PSO with the fitness value of 0.0062. The results of fitness curve for SVM-GA obtained MSE value of 
0.0065. The parameters of cost and gamma gives by SVM-PSO are 3.0522 and 0.2471. As for SVM-GA, 
the value of parameters achieved are 2.0638 and 0.3318 for cost and gamma. This shows that the fitness 
values obtained from SVM-PSO is slightly better as compared to SVM-GA. 

Figure 7 shows the comparison graph between SVM-PSO, SVM-GA and SVM-Grid search using 
the parameters obtained earlier. From the graph during training phase, it shows that SVM-GA obtained 
the highest correlation coefficient followed by SVM-PSO and SVM-Grid search. The values of correlation 
coefficient obtained are 98.19%, 97.83% and 96.17% with the corresponding MSE values of 0.0032, 0.0040 
and 0.0041 respectively. From the result, it can be seen that all three method achieved a very well prediction 
as it is higher than 90% in predicting concentration COD effluent. Meanwhile during testing phase, 
the testing results shows SVM-PSO achieved the best prediction result with correlation coefficient value of 
94.59% and MSE value of 0.0064. It followed by SVM-GA with R’ of 94.11% and MSE value of 0.0067. 
SVM-Grid search obtained the lowest prediction as the values achieved are 93.43% and 0.0070 for R* and MSE. 
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Figure 7. Comparison graph between measured and predicted of SVM-PSO, SVM-GA and SVM-Grid search 
for training and testing data 
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The overall performance comparison for training and testing results is shown in Table 2. From 
the results obtained, it clearly shown that with the optimization method could enhance the results instead of 
using conventional way of grid search method. The hybridization of SVM-PSO achieved the highest 
performance during testing dataset followed by SVM-GA and SVM-Grid Search. 


Table 2. Overall performance for SVM-PSO, SVM-GA and SVM-Grid search model 


Training Testing 
ner Moge! C Y R2 (%) MSE R2 (%) MSE 
50°C SVM-PSO 3.0522 0.2471 97.83 0.0040 97.83 0.0040 
SVM-GA 2.0638 0.3318 98.19 0.0032 98.19 0.0032 
SVM-Grid Search 4 0.125 96.17 0.0041 96.17 0.0041 


5. CONCLUSION 

This paper presents the modelling and optimization techniques for AGS using SVM hybrid 
with PSO, GA and grid search. The SVM computational approach has been widely considered by 
a researchers due to its great ability in finding the local minimum, solved a non-linearity problems, and can 
be applied in high dimension system. This can be proven from the simulation results, it can be observed that 
the testing result using optimization method such as PSO and GA could produces better accuracy of 
the model prediction at the temperature of 50°C. As compared to conventional method which is grid search, 
it is time-consuming are not able to obtained the best local minima in determine the parameters. This provide 
and evident that SVM-PSO and SVM-GA can give better prediction compared to SVM-Grid search model 
for limited training data. Basically, three major advantages of SVM computational approach can be highlight 
through this research. First, only three type of kernel parameters needed to be determined which are kernel 
function, cost of penalty and gamma value. Second, SVM itself are unique, optimal and global in solving 
quadratic problem. Third, SVM is well known with good generalization performance due to 
the implementation of SRM principle. For the recommendation of the future work, to improve 
the computational time of searching the SVM parameters in future, external optimization such as 
gravitational search algorithm (GSA) can be used as it was reported to give faster convergent and reliable to 
meet the objective function in many types of application. Hence, in SVM, GSA is a very potential 
optimization that can be considered. 
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